1. Field of the Invention
The present invention relates to a video server apparatus capable of performing various kinds of special playbacks of coded moving images more smoothly.
The invention also relates to a video-on-demand system, a broadcasting system, a video library system, and the like which consist of a video server apparatus for storing moving image programs and terminal apparatuses for displaying images supplied from the video server, and particularly to a technique for enabling a high-speed playback of moving images at an arbitrary speed.
2. Description of the Related Art
A video server apparatus has entered a commercial stage which apparatus stores a number of compressed/coded moving image programs in a storage medium such as a hard disk and performs simultaneous, multiple playbacks of the same or different image programs in response to requests from a plurality of terminal apparatuses. A video-on-demand system has been realized by combining the above server apparatus and terminal apparatuses in which system the server apparatus immediately responds to individual program requests from terminal apparatuses. In museums etc., a video library system utilizing such a video-on-demand system is constructed to allow visitors to conduct in-house perusal. Further, CATV broadcasting system companies are now considering a service for providing, by utilizing a video-on-demand system, a movie that is selected by a user at his terminal apparatus. In this specification, all of the above systems are generally referred to as a "video-on-demand system."
In the above systems, not only a normal playback but also various kinds of special playbacks such as a fast-forward playback and a rewind playback are attracting great interest. For example, reference is made to Y. Ishibashi et al.: "Multiple-access Visual Search for Digital Video-on-demand Services," technical Report of IEICE, CS92-74, IE92-96, pp. 101-106, December 1992, and T. Mori et al.: "Video Storage and Visual Search techniques in VOD System," Technical Report of IEICE, IE94-81, pp. 9-16, November 1994.
A conventional video-on-demand system will be described below. FIG. 1 is a block diagram showing its configuration. In FIG. 1, a video server apparatus VS consists of a hard disk 1 as a storage medium, AV (audio-video) streams 1a, an access table 1b, a stream reading section 2; a stream transmitting section 3; an access table managing section 4; a command receiving section 5, and a stream control section 6. A terminal apparatus ST consists of a stream receiving section 7, a command transmitting section 8, a decoder section 9, and an input control section 10. A monitor 11, a speaker 12, and an input device 13 are connected to the terminal apparatus ST.
FIG. 2 is a block diagram showing details of the stream control section 6 of the above conventional video server apparatus VS. In FIG. 2, reference numeral 81 denotes a stream readout position determining section and numeral 82 denotes a keyframe skip number setting table.
FIG. 3A illustrates keyframe readout intervals of a high-speed playback in the above server apparatus. In FIG. 3A, reference numerals 41-43 denote keyframes that are read out first, second, and third, and characters a-c denote readout periods of the keyframes 41-43 (constant read bit rate).
FIG. 4A illustrates keyframe readout intervals of another type of high-speed playback in the above server apparatus. In FIG. 4A, reference character n means time taken to read out a storage unit block at a constant bit rate. Reference numerals 51-53 denote keyframes that are read out first, second, and third, and symbols a'-c' denote readout periods of the keyframes 51-53 when they are read out at a constant bit rate on a storage unit block basis.
FIG. 5A illustrates the operation of the access table managing section 4 of the above server apparatus. In FIG. 5A, reference character u means time taken to read out the access table 1b at a constant bit rate.
FIG. 6A illustrates a frame storage order of an AV stream 1a, and FIG. 6B illustrates a frame playback order in a normal playback.
The above-configured video server apparatus operates in the following manner. A signal that is input from the input device 13 is converted by the input control section 10 into commands indicating a playback start position and a playback speed, which are transmitted from the command transmitting section 8 to the command receiving section 5. The stream control section 6 determines keyframes to be read out and their storage locations based on the playback start position and the playback speed that are received by the command receiving section 5 and keyframe addresses that are managed by the access table managing section 4, and gives the determined information to the stream reading section 2. More specifically, in the stream control section 6, the stream readout position determining section 81 determines keyframes to be read out based on the playback speed that is obtained from the command receiving section 5, the keyframe interval, and the coding bit rate of the AV stream 1a, and also determines readout positions of the AV stream 1a based on the playback start position that is obtained from the command receiving section 5 and keyframe addresses that are managed by the access table managing section 4. The determined information is sent to the stream reading section 2. The stream reading section 2 reads out the AV stream 1a at a designated read bit rate, and a read-out stream is transmitted from the stream transmitting section 3 to the stream receiving section 7. The stream received by the stream receiving section 7 is decoded by the decoder section 9, and output as images and sound from the monitor 11 and the speaker 12, respectively.
An AV stream 1a, which has been obtained by compressing/coding an audio-associated video signal according to the international standard MPEG (moving picture experts group), are stored in a frame order shown in FIG. 6A. In the case of a normal playback, the stream reading section 2 reads out all of the AV stream 1a, the stream transmitting section 3 transmits all of the read-out data, and the data are output from the monitor 11 in a frame order shown in FIG. 6B. In the case of a fast-forward or rewind playback, the stream reading section 2 reads out only keyframes, i.e., I frames, the stream transmitting section 3 transmits only the keyframes, and the keyframes are output from the monitor 11 in a frame order of FIG. 3A, FIG. 4A, or the like.
For convenience of description, frame types of I, P and B and display frame order numbers of a normal playback are indicated in FIGS. 3A, 4A and 6A. Frame type I corresponds to an intra-coded image that has been subjected to intraframe coding, frame type P corresponds to a forward prediction coded image that has been subjected to motion-compensation interframe coding, and frame type B corresponds to a bidirectional prediction coded image that has been subjected to motion-compensation interframe coding.
The access table 1b has descriptions of a head address of each I frame of an AV stream 1a, a data length of each I frame, a coding bit rate of the AV stream 1a, and other items, and is managed by the access table managing section 4.
However, the above conventional video server apparatus has a problem that where the data length of a keyframe varies stream by stream, a constant playback speed is not obtained in reading streams at a constant bit rate in a high-speed playback even if the same keyframe skip number is set.
The above conventional video server apparatus has another problem that where the data length of a keyframe varies in one stream, the time taken to read or transmit one keyframe varies in reading or transmitting a stream at a constant bit rate in a high-speed playback.
The above conventional video server apparatus has still another problem that where the data length of a keyframe varies in one stream, the time taken to read or transmit one keyframe varies in reading or transmitting a stream at a constant bit rate on a fixed-length block basis in a high-speed playback.
Further, in the above conventional video server apparatus, since the entire access table is acquired immediately before a playback at a constant bit rate, a response time from a stream playback request to a start of an actual playback of the requested stream is long.
FIG. 7 shows another conventional video-on-demand system. As shown in FIG. 7, a video server apparatus of this system consists of a hard disk 101 for storing a number of AV streams 101a, a hard disk 102 for storing an access table 102b to be used in accessing the AV streams 101a, a stream reading section 103 for reading out an AV stream 101a from the hard disk 101, a stream transmitting section 104 for transmitting the readout AV stream to a terminal apparatus, an access table managing section 105 for managing the access table 102b, a command receiving section 106 for receiving a command from the terminal apparatus, a stream control section 107 for controlling reading and transmission of the AV stream in response to a request from the terminal apparatus, and a keyframe skip number setting table 108 in which keyframe readout intervals of high-speed playbacks are set.
The terminal apparatus consists of an input control section 113 for generating a command in response to an input manipulation on an input device 116, a command transmitting section 109 for transmitting the generated command to the video server apparatus, a stream receiving section 110 for receiving an AV stream 101a from the video server apparatus, a decoder section 112 for decoding the received AV stream 101a, and a decoder control section 111 for controlling the decoding and playback timing of the decoder section 112. The input device 116 such as a keyboard or a mouse, a monitor 114 for displaying decoded images and a speaker 115 for outputting decoded sound are connected to the terminal apparatus.
In this video-on-demand system, a signal that is input through the input device 116 and indicates a designated program, a playback start position, a playback speed, etc. is converted by the input control section 113 into a program designation command and commands indicating a playback start position and a playback speed of moving images, which commands are transmitted from the command transmitting section 109 to the command receiving section 106 of the video server apparatus.
The stream control section 107 selects an AV stream 101a of the designated program from among a number of AV streams 101a stored in the hard disk 101 and determines the storage location of a keyframe located in the vicinity of the designated playback start position based on the program designation information, the playback start position, and the playback speed that have been received by the command receiving section 106, and keyframe information and a coding bit rate of the AV stream 101a which are managed by the access table managing section 105. The stream control section 107 further determines a read bit rate based on the coding bit rate etc. The stream control section 107 gives the thus-determined information to the stream reading section 103 and instructs it to start reading out the selected stream 101a.
In a normal playback, the stream control section 107 sets the read bit rate equal to the coding bit rate. In a high-speed playback, the stream control section 107 sets the read bit rate equal to or lower than the coding bit rate.
If the playback speed obtained from the command receiving section 106 means a high-speed playback, the stream control section 107 determines keyframes to be read out in the high-speed playback and a read bit rate based on the received playback speed, the relationship between the playback speed multiplication factor and the keyframe readout interval which is described in the keyframe skip number setting table 108, and the information managed by the access table managing section 105. The stream control section 107 then instructs the stream reading section 103 to read out the keyframes thus determined, and to add a playback mode indicating a change in playback speed to a read-out stream.
The stream reading section 103 reads out the designated keyframes from the AV stream 101a at the read bit rate that is designated by the stream control section 107, and inserts or buries a playback mode into the read-out stream that consists only of the keyframes.
The stream transmitting section 104 transmits the stream that is received from the stream reading section 103 to the stream receiving section 110 of the terminal apparatus. At this time, the stream control section 107 controls the transmission bit rate in accordance with the number of terminal apparatuses currently communicating with the video server apparatus and other information.
In the terminal apparatus, the decoder control section 111 controls the decoding and playback timing of the decoder section 112 by using, as a trigger, the playback mode extracted from the stream by the stream receiving section 110. Controlled in this manner, the decoder section 112 decodes the stream received by the stream receiving section 110. The decoded stream is output from the monitor 114 as images, and also output from the speaker 115 as sound in the case of a normal playback.
In the case of a normal playback, the entire AV stream 101a that is stored in the hard disk 101 in a frame order shown in FIG. 6A is read out by the stream reading section 103, and then transmitted by the stream transmitting section 104. The stream 101a is output from the monitor 114 in a frame order shown in FIG. 6B.
In the case of a high-speed playback, the stream reading section 103 reads out only the designated keyframes (i.e., I frames), the stream transmitting section 104 transmits the read-out keyframes, and the monitor 114 continues to display each of the transmitted I frames over a given period as shown in FIGS. 8A-8D. In Figs. BA-8D, solid-line frames are frames that are actually decoded and broken-line frames are frames that are a previously decoded frame which is kept displayed as it is. FIG. 8A shows a case where I frames are read out in the order of I(1), I(16), I(31), I(46), . . . . (the parenthesized numbers indicate a display order of frames in a normal playback) and each of the I frames is displayed on the monitor 114 over a playback period of 9 frames. Similarly, FIG. 8B shows a case where I frames are read out in the order of I(1), I(31), I(61), I(91) , . . . and each of the I frames is displayed on the monitor 114 over a playback period of 9 frames.
In the case of FIG. 8A, images that would have a frame interval of 15 frames in a normal playback are played back over a frame interval of 9 frames (images are updated every 9 frames). Therefore, the playback speed multiplication factor is 15/9=1.67. In the case of FIG. 8B, images that would have a frame interval of 30 frames in a normal playback are played back over a frame interval of 9 frames. Therefore, the playback speed multiplication factor is 30/9=3.33.
FIG. 9 is a playback speed table showing a relationship between a playback speed multiplication factor and a combination of a keyframe readout interval and a keyframe playback interval. This table relates to a case where the keyframe interval, which is the number of frames existing between adjacent keyframes in a stream, is 15. The horizontal axis represents a keyframe readout interval x, that is, a frame interval between actually read-out keyframes of an AV stream 101a, and the vertical axis represents a keyframe playback interval y, that is, a frame interval during which the terminal apparatus continues to play back the same keyframe. A playback speed, which is calculated as x/y, is written at an intersection of x and y.
In the conventional video server apparatus, the keyframe skip number setting table 108 describes, in accordance with the system specification, a relationship corresponding to part of the playback speed table of FIG. 9, for instance, a relationship between the frame readout interval x and the playback speed multiplication number in a case where the keyframe playback interval is 9 (region 191 in FIG. 9).
When a playback speed of a multiplication factor 2, 3, 4 or 5 is designated by the terminal apparatus, the stream control section 107 of the video server apparatus selects a playback speed that is closest to the designated speed by referring to the relationship described in the keyframe skip number setting table 108, and determines a keyframe readout interval x that corresponds to the selected playback speed. In this example, when a double-speed playback is designated by the terminal apparatus, a keyframe readout interval x of 15 corresponding to a playback speed multiplication factor 1.67 is selected. When a triple-speed playback is designated, a keyframe readout interval x of 30 corresponding to a multiplication factor 3.33 is selected. When a quadruple-speed playback is designated, a keyframe readout interval x of 30 corresponding to a multiplication factor 3.33 is also selected. When a 5-fold-speed playback is designated, a keyframe readout interval x of 45 corresponding to a multiplication factor 5 is selected. The stream reading section 103 is so controlled as to read out I frames of the AV stream 101a from the hard disk 101 at the keyframe readout interval x thus determined. In this example, the actual playback speed multiplication factors are 1.67, 3.33, 3.33 and 5 for the designated multiplication factors of 2, 3, 4 and 5, respectively.
According to the system specification, the terminal apparatus continues to display each keyframe that is transmitted from the video server apparatus while setting the keyframe playback interval y at 9. FIGS. 8A-8D show playback frame sequences in the terminal apparatus for designated playback speed multiplication factors of 2, 3, 4 and 5, respectively.
However, in the conventional video-on-demand system, even if an arbitrary playback speed multiplication factor is designated by the terminal apparatus, an actual playback is performed with a multiplication factor that is an approximation of the designated one. Further, there may occur an event that actual playbacks are performed at the same speed even for designated playback speeds that are different from each other, as in the case of triple-speed and quadruple-speed playbacks in the above example. As such, the actual playback speed cannot be controlled finely.
Further, there is a possibility that because the keyframe readout interval varies stream by stream, a designated playback speed cannot be attained without changing the keyframe playback interval. However, in this case, since the conventional system does not have any means for informing the terminal apparatus of a keyframe playback interval that has been determined for reasons on the video server apparatus side, the terminal apparatus cannot control the keyframe playback interval in accordance with the characteristics of a stream. As a result, a high-speed playback is performed at an improper speed.